Systems and methods for recording haptic data for use with multi-media data

ABSTRACT

A system includes a recorder configured to record audio and/or video of a subject of interest and output a recording of the subject of interest and a non-contact sensor associated with the recorder. The non-contact sensor is constructed and arranged to measure movement and/or vibration of the subject of interest from substantially the same perspective and at the same time as the recorder. The system includes a controller configured to transform the measured movement and/or vibration of the subject of interest measured by the non-contact sensor into a tactile data stream for sending to a haptic display device for playback with the recording of the subject of interest by the recorder and providing haptic effects corresponding to the measured movement and/or vibration to a user of the haptic display device in synchronization with the recording.

FIELD

The present invention is directed to systems and methods for recordinghaptic data for use with multi-media data.

BACKGROUND

In both live and pre-produced content creation, there arewell-established techniques for setting up and capturing action. In thecase of a location based live sporting event, for example, a siteproducer typically arrives one to two weeks in advance of the event tosetup a variety of camera positions in order to create a compelling liveproduction (broadcast) stream of multi-media data from the location. Itis normal for a site production crew to pre-mix a set of on-locationmicrophones to create an “egocentric” sound field for each cameraposition to enrich the experience for recipients of the broadcastedcontent.

For example, at a sports stadium, cameras that are mounted high abovethe stands may include both crowd sound effects and on-field soundeffects mixed equally. Field-side cameras can have parabolic or focusedmicrophones to create a greater sense of on-field presence. Athletemounted cameras may have athlete mounted microphones and be mixed withfield-side cameras to provide an athlete-centric and stadium experience.Although similar strategies are used for pre-produced content for moviesand television, there is significantly more audio post-productionediting that is done to create the final product.

SUMMARY

It is desirable to create a more immersive experience for audiencesviewing live sports events and other entertainment on electronic devicesby adding haptic content to the broadcast that can be streamed andplayed back to users of the electronic devices. It is also desirable tomake the haptic content more pervasive by making real-world haptic dataeasier to obtain and transforming the real-world haptic data with aslittle human intervention as possible in order to facilitate propagationand user acceptance of the experience.

According to an aspect of the invention, there is provided a system thatincludes a recorder configured to record audio and/or video of a subjectof interest and output a recording of the subject of interest, and anon-contact sensor associated with the recorder. The non-contact sensoris constructed and arranged to measure movement and/or vibration of thesubject of interest from substantially the same perspective and at thesame time as the recorder. The system includes a controller configuredto transform the measured movement and/or vibration of the subject ofinterest measured by the non-contact sensor into a tactile data streamfor sending to a haptic display device for playback with the recordingof the subject of interest by the recorder and providing haptic effectscorresponding to the measured movement and/or vibration to a user of thehaptic display device in synchronization with the recording.

In an embodiment, the non-contact sensor includes a laser vibrometer. Inan embodiment, the non-contact sensor includes a tracking device.

In an embodiment, the non-contact sensor includes video stream analysisalgorithms.

In an embodiment, the recorder and the non-contact sensor areco-located.

In an embodiment, the system includes a first servomechanism that iscontrolled by the controller and is constructed and arranged to move thenon-contact sensor to a position and/or orientation having substantiallythe same perspective as the recorder. In an embodiment, the systemincludes a second servomechanism that is controlled by the controllerand is constructed and arranged to move the recorder. The firstservomechanism is synchronized with the second servomechanism so thatmovement of the non-contact sensor is synchronized with movement of therecorder.

In an embodiment, the system includes a second sensor constructed andarranged to be in contact with the subject being recorded by therecorder and to sense movement and/or vibration of the subject. Thecontroller is further configured to combine data received from thenon-contact sensor and the second sensor and transform the combined datainto the tactile data stream. In an embodiment, the controller includesa transformer constructed and arranged to mix sensors signals havingdifferent sampling rates and/or sensitivity and output the transformedcombined data.

According to an aspect of the invention, there is provided a method thatincludes recording audio and/or video of a subject of interest with arecorder, and measuring movement and/or vibration of the subject ofinterest with a non-contact sensor associated with and havingsubstantially the same perspective as the recorder. The method includestransforming, with a controller, the measured movement and/or vibrationinto a tactile data stream for sending to a haptic display device forplayback with the recording of the audio and/or video of the subject ofinterest and providing haptic effects corresponding to the measuredmovement and/or vibration to a user of the haptic display device insynchronization with the recording.

In an embodiment, the method includes moving the non-contact sensor,with a first servomechanism controlled by the controller, to a positionhaving substantially the same perspective as the recorder as therecorder moves and changes perspective. In an embodiment, the methodincludes moving the recorder, with a second servomechanism controlled bythe controller. The first servomechanism is synchronized with the secondservomechanism so that movement of the non-contact sensor issynchronized with movement of the recorder.

In an embodiment, the method includes sensing movement and/or vibrationof the subject of interest with a second sensor in contact with thesubject of interest, combining data received from the non-contact sensorand the second sensor, and transforming the combined data into thetactile data stream.

In an embodiment, the method includes transforming sensor signals havingdifferent sampling rates and/or sensitivity and outputting thetransformed combined data.

These and other aspects, features, and characteristics of the presentinvention, as well as the methods of operation and functions of therelated elements of structure and the combination of parts and economiesof manufacture, will become more apparent upon consideration of thefollowing description and the appended claims with reference to theaccompanying drawings, all of which form a part of this specification.It is to be expressly understood, however, that the drawings are for thepurpose of illustration and description only and are not intended as adefinition of the limits of the invention. As used in the specificationand in the claims, the singular form of “a”, “an”, and “the” includeplural referents unless the context clearly dictates otherwise.

BRIEF DESCRIPTION OF THE DRAWINGS

The components of the following Figures are illustrated to emphasize thegeneral principles of the present disclosure and are not necessarilydrawn to scale. Reference characters designating correspondingcomponents are repeated as necessary throughout the Figures for the sakeof consistency and clarity.

FIG. 1A schematically illustrates a system in accordance with anembodiment of the invention;

FIG. 1B schematically illustrates further details of a controller of thesystem of FIG. 1A;

FIG. 2A schematically illustrates an implementation of the system ofFIG. 1 in accordance with an embodiment of the invention;

FIG. 2B schematically illustrates further details of the embodiment ofthe invention illustrated by FIG. 2A;

FIG. 3 schematically illustrates data flow from a non-contact sensor anda video/audio recorder to a playback device in accordance with anembodiment of the invention;

FIG. 4 schematically illustrates data flow from a non-contact sensor, acontact sensor and a video/audio recorder to a playback device inaccordance with an embodiment of the invention; and

FIG. 5 schematically illustrates a method in accordance with anembodiment of the invention.

DETAILED DESCRIPTION

FIG. 1A schematically illustrates a system 100 in accordance with anembodiment of the invention. As illustrated, the system 100 includes oneor more sensors 102, which are configured to sense movement and/orvibrations experienced by a subject of interest of an event from adistance, and convert the sensed movement and/or vibrations into tactileeffect data for playback to a user of an electronic playback device 130as haptic effects, as described in further detail below. In anembodiment, the sensor 102 is a non-contact sensor, as described infurther detail below.

In an embodiment, the sensor 102 is constructed and arranged to sensethe motion and/or vibration of players and/or equipment or other subjectof interest as part of a sporting event, etc. In an embodiment, ambientmotion and vibration on a playing field/rink/court and boundaries of theplaying field/rink/court, etc., may be sensed by the sensor 102.Embodiments of the sensor 102 are described in further detail below. Thevibration that is sensed by the sensor 102 may be in any frequencyrange, even if the vibration frequency range may not be normally felt byhuman mechanoreceptors. In such an embodiment, the vibration may then beconverted or translated into the mechanoreceptive range during thetransformation of the sensor data. Transformation of the sensor datainto tactile effect data is described in further detail below.

The system 100 also includes one or more cameras 104 constructed andarranged to capture images and sounds of the subject of interest. Thecamera 104 may include an integral video/audio recorder or may includeseparate video and audio recorders, denoted by 106 and 108,respectively, in FIG. 1A. The audio recorder 108 may include one or moremicrophones constructed and arranged to capture the sounds associatedwith the event. For example, if the event is a football game, one ormore microphones may be placed at field level on a sideline of theplaying field and one or more microphones may be placed at locationsabove the playing field, as is currently done during broadcasts offootball games, for example. As used herein, “camera” refers to thecombination of the video recorder 106 and the audio recorder 108, evenif the two recorders are not co-located and are physically separate. Inan embodiment, the sensor(s) 102, the video recorder 106, and the audiorecorder 108 may be part of the same electronic device. In anembodiment, the sensor(s) 102, the video recorder 106, and the audiorecorder 108 may be separate, stand-alone devices or part of separate,stand-alone devices.

As illustrated in FIG. 1A, the system 100 also includes a controller 110constructed and arranged to output control signals to the sensor(s) 102,the video recorder 106 and the audio recorder 108 and to also receivedata signals from the sensor(s) 102, the video recorder 106 and theaudio recorder 108 for further processing and ultimate transmission tothe electronic playback device 130. The controller 110 includes one ormore processors 112 constructed and arranged to process signals and dataoutput by the sensor(s) 102, the video recorder 106, and the audiorecorder 108. The processor(s) 112 may be programmed to execute computerprogram modules comprising executable instructions for carrying outvarious functions of the controller 110, as described in further detailbelow. In an embodiment, the sensor(s) 102 may include processor(s) thatare separate from the processor(s) 112 or the processor(s) 112 may bepart of the sensor(s).

The electronic playback device 130 includes a haptic output device 132configured to output haptic effects to a user of the electronic playbackdevice 130, a display 134 configured to display images, such as theimages captured by the video recorder 106, and a speaker 136 configuredto output sound, which may be the sound captured by the audio recorder108. In an embodiment, the haptic output device 132 may be part of awearable device, such as a bracelet, watch, arm band, head band, headmounted display, etc. In an embodiment, the haptic output device 132,the display 134, and the speaker 136 may be separate devices that areconfigured to communicate with each other through a wireless connection,for example. In an embodiment, the haptic output device 132 may be partof a wearable device, the display 134 may be part of a television, andthe speaker 136 may be a wireless speaker that is separate from thedisplay 134.

The haptic output device 132 may include an actuator, for example, anelectromagnetic actuator such as an Eccentric Rotating Mass (“ERM”) inwhich an eccentric mass is moved by a motor, a Linear Resonant Actuator(“LRA”) in which a mass attached to a spring is driven back and forth,or a “smart material” such as piezoelectric, electro-active polymers orshape memory alloys, a macro-composite fiber actuator, an electro-staticactuator, an electro-tactile actuator, and/or another type of actuatorthat provides a physical feedback such as a haptic (e.g., vibrotactile)feedback. The haptic output device 132 may include non-mechanical ornon-vibratory devices such as those that use electrostatic friction(ESF), ultrasonic surface friction (USF), or those that induce acousticradiation pressure with an ultrasonic haptic transducer, or those thatuse a haptic substrate and a flexible or deformable surface, or thosethat provide projected haptic output such as a puff of air using an airjet, and so on.

FIG. 1B illustrates further details of an embodiment of the controller110. The controller 110 may be a general-purpose or specific-purposemicrocontroller for managing or controlling the operations and functionsof the system 100. For example, the controller 110 may be specificallydesigned as an application-specific integrated circuit (“ASIC”) tocontrol output signals to the haptic output device 132 to provide hapticeffects. The controller 110 may be configured to decide, based onpredefined factors, what haptic effects are to be generated, the orderin which the haptic effects are generated, and the magnitude, frequency,duration, and/or other parameters of the haptic effects. The controller110 may also be configured to provide streaming commands that may beused to drive the haptic output device 132 for providing a particularhaptic effect. In an embodiment, the controller 110 may include morethan one processor 112, i.e. a plurality of processors 112, eachconfigured to perform certain functions within the system 100. Thecontroller 110 may also include electronic memory that includes one ormore storage devices that may include haptic effect profiles,instructions for how the haptic output device 132 is to be driven,and/or other information for generating haptic effects. In anembodiment, the electronic memory 114 may be part of the processor 112.

The electronic memory 114 may be used to store data sensed by thesensor(s) 102, data that is recorded by the video recorder 106, and datathat is recorded by the audio recorder 108. The electronic memory 114may include one or more internally fixed storage units, removablestorage units, and/or remotely accessible storage units. The variousstorage units may include any combination of volatile memory andnon-volatile memory. The storage units may be configured to store anycombination of information, data, instructions, software code, etc.

As illustrated, the controller 110 also includes an input transformer116, a sensor data encoder 118, a video/audio data encoder 120, adecoder 122, and an output transformer 122. As discussed above, the datagenerated by the sensor(s) 102 may be stored in the electronic memory114. In addition, the data generated by the sensor(s) 102 may betransformed by the input transformer 116 prior to being stored in theelectronic memory 114. The transformation of the sensor data isconsidered to be an optional step and whether the transformation isneeded may depend on the nature of the sensors being used. For example,in embodiments that use multiple sensors 102 that each use a differentfrequency for sampling, etc., the input transformer 116 may be used totransform the data such that the data may be combined into a single datastream having the same frequency, etc.

In an embodiment, the sensor data encoder 118 may be configured to placethe sensor data during recording into a flexible container format, suchas an MPEG-4 file, that allows for the storage of data other than videoand audio in a single file container. Similarly, the video/audio dataencoder 120 may be configured to place the video and audio data duringrecording into a container format, such as an MPEG-4 file. In anembodiment, software may be written to store the sensor data in aseparate file, but with special markers in the sensor data to allow forproper synchronization with the video/audio data at playback time. Insuch an embodiment, very little input transformation may need to beapplied, beyond shaping the sensor data to conform to the limitations ofthe designed recording format. The exact format may be determined by theimplementer. Once the user recording the event has completed his or heractivity, the recording may be stopped. The MPEG-4 file may be closed,and all of the sensor data may reside in the MPEG-4 file in theelectronic memory 114 with the video/audio data.

The decoder 122, which may be part of the controller 110 as illustratedor may be part of a media player of the electronic playback device 130that is configured to playback the media file, is configured to read thedata generated by the sensor(s) 102 from the electronic memory 114, andassociate the data temporally with the audio data and video data thatwere recorded and stored in the electronic memory 114. During mediaplayback, the decoder 122 may pass the sensor data through an outputtransformer 124 configured to transform the sensor data into a hapticoutput signal to generate one or more haptic effects or haptic sensorycommands, which include but are not limited to, vibration, surfacefriction modulation, skin pinch, skin squeeze, etc. The decoder 122 maybe configured to synchronize the haptic output signal that wastransformed from the sensor data with the video/audio data so that thehaptic effect is synchronized with the video and audio during playback.In an embodiment, the synchronization may be completed by ensuring thattime is the same in the video data, the audio data, and the hapticeffect during playback.

The haptic output signal may then be transmitted from the decoder 122 tothe haptic output device 132 so that the person(s) experiencing themedia through the electronic playback device 130 that includes thehaptic output device 132 may more fully experience the event beingplayed back. The electronic playback device 130 may be any device, suchas an electronic handheld device, such as a mobile phone, gaming device,personal digital assistant (“PDA”), portable e-mail device, portableInternet access device, tablet, etc. The electronic playback device 130may include, but is not limited to, a handheld device or wearable devicewith the display 134, which may be a high definition display, thatdisplays the media, and a handheld object that is capable of producinghaptic sensations or effects, or an object attached to the user's body,leaning up to the user's body, or otherwise able to transmit tactilesensations and haptic effects to the user. The synchronization of all ofthe data streams containing vibration, video, and audio data may bemanaged by recording software, which may reside in the processor(s) 112of the system 100 illustrated in FIGS. 1A and 1B.

Either at a later time, or concurrently with the activity beingperformed, one or more viewers may be interested in experiencing theactivity. To playback the activity, the viewer may launch theappropriate playback software on their individual electronic playbackdevice 130. In an embodiment, the playback software may include a playersoftware application that incorporates the sensor decoding schemeperformed by the decoder 122 described above, as well as outputtransform software that may be run by the output transformer 124, inorder to transform the sensor data into a haptic output signal suitablefor the haptic output device 132 in the electronic playback device 130.In an embodiment, a player software application may incorporate thesensor decoding scheme. The player software may rely on the outputtransform software being resident or otherwise pre-installed on theelectronic playback device 130, and such output transform software maytransform the sensor data into the haptic output signal suitable for thehaptic output device 132 in the electronic playback device 130. In otherwords, the output transformer 124 and/or decoder 122 may be located onthe electronic playback device 130.

In an embodiment, the electronic playback device 130 may be configuredto decode the stream of video and tactile effect data, and maintain thesynchronization between the video and the tactile effect data. A singlemicrocontroller in the electronic playback device 130 may, for example,control both the video display 134 and the haptic output device 132based on the video and tactile effect data streams. In an embodiment, aplayer software application may rely on the playback device's operatingsystem software to perform the media playback, which incorporates thesensor decoding scheme. The operating system software may rely on theoutput transform software being resident or otherwise pre-installed onthe electronic playback device 130, such output transform softwaretransforming the sensor data into a haptic output signal suitable forthe haptic output device 132 in the electronic playback device 130. Theviewer may then experience haptic effects associated with the viewing ofthe event or performance, such haptic effects being produced by theoutput transform software. In an embodiment, the output transformationmay occur outside of the electronic playback device 130 and instead mayoccur on server(s) owned by broadcaster(s) and stored in the cloud. Insuch an embodiment, multiple versions of the haptic track may begenerated for different devices and/or device types and the appropriatehaptic track may be selected and transmitted to the haptic output device132 in the electronic playback device 130.

In an embodiment, the video, audio and sensor data streams may besynchronized, merged, and transmitted to the playback device 130. Thesynchronization may, for example, be done by including a timestamp onevery video frame and sensor measurement, keeping in mind that thecapture may take place on independent devices that communicate through awired or wireless network. The recording device may therefore need toobtain a shared time reference, for example from a GPS system. In anembodiment, synchronization may be performed by performing a specificaction that is detectable in both the video and the sensor data streams.

The resulting data may be transmitted as a single data stream combiningvibrations and video, which also includes audio, or two data streamswith synchronization information. The data stream may be transmittedgradually to the electronic playback device 130, or stored in a file forlater playback. In an embodiment, the haptic effects may be producedoffline using editing tools and added to the video in post-production.

The video may be played back at a later time or streamed in real time,in both cases either to one or more recipients. The video may be playedback on several devices, including but not limited to smartphones ortablets, computers, home theater systems, etc. The haptic effects maysimilarly be produced using different haptic output devices 132 locatedon a smartphone or a tablet, a wearable device, such as a head mounteddisplay device, a wristband, a ring or a glove, or a piece of furniture,such as a chair or a desk.

FIG. 2A schematically illustrates a system 200 that is an embodiment ofthe system 100 described above with respect to FIGS. 1A and 1B. In theembodiment illustrated in FIG. 2A, the sensor(s) 102 may include amovement/vibration capture device 210, which may include a contactlessmeasurement device 212 constructed and arranged to measure movementand/or vibrations of the subject of interest and/or a tracking device214 constructed and arranged to measure movement and/or vibrations ofthe subject of interest. Although both the contactless measurementdevice 212 and the tracking device 214 are illustrated as being part ofthe movement/vibration device 210, in some embodiments, themovement/vibration device 210 may include only one of the contactlessmeasurement device 212 or the tracking device 214. The system 200 alsoincludes a video/audio recorder 220, a transformer 230 and an electronicplayback device 240, which includes at least one haptic output device242 configured to generate haptic effects HE representing the movementand/or vibrations captured by the movement/vibration capture device 210,and a display 244 configured to display the images captured by thevideo/audio recorder 220.

In an embodiment, the contactless measurement device 212 may be a laservibrometer that is configured to measure the vibration of the subject ofinterest. As is known in the art, a laser vibrometer, in use, emits alaser beam at a first frequency directed towards the subject of interestand measures a second frequency of the laser beam that is reflected bythe subject of interest and detected by a photodetector. If the surfaceof the subject is moving, e.g. vibrating, there will be a shift in thefrequency such that the second frequency is different from the firstfrequency. The shift in frequency can then be correlated to thevibration of the subject of interest. The tracking device 214 may beconfigured to track movement of the subject of interest in up to threedimensions so that vibrations of the subject of interest can bedetermined via image analysis by known methods. In an embodiment, themovement and/or vibration may be derived from a two dimensional or threedimensional video stream using image analysis software.

In an embodiment, one or more additional sensors 216 configured to sensemovement and/or vibrations, such as an accelerometer, may be placed onthe subject of interest. Such sensor(s) 216 are different from thenon-contact sensor(s) described herein in that they are in contact withthe subject instead of not being in contact with the subject ofinterest.

In an embodiment, one or more additional sensors (not shown) may includeoptical sensors configured to track a change in the light fieldsurrounding the subject of interest in a particular range, and a changein the light field may be used as the sensor data indicative ofmovement/vibration.

As also illustrated in FIG. 2A, the video/audio recorder 220, which maybe a camera, may be implemented as the video recorder 106 and audiorecorder 108 described above with respect to FIG. 1A and configured tocapture images of and sounds from the subject of interest. Although thevideo/audio recorder 220 is illustrated as an integral device,embodiments of the invention cover arrangements in which the videorecorder is separate from the audio recorder, as described above. In anembodiment, the movement/vibration capture device 210 and thevideo/audio recorder 220 may be co-located on the same electronicdevice.

The data captured by the movement/vibration capture device 210 and thecontact sensor 216 may be processed by the transformer 230 and output astactile effect data for transmission to the electronic playback device240. In an embodiment, the tactile effect data may be stored alongside,or interleaved with, audio-visual data for later rendering or forimmediate broadcast for live event spectator consumption.

In an embodiment, sensor data fusion techniques, such as Kalmanfiltering may be used by the transformer 230 if the different sensors212, 214, 216 use different sampling rates and/or have differentsensitivity, for example. The tactile effect data that is output by thetransformer 230 may be encoded prior to transmission to the electronicplayback device 240 and decoded upon playback on the electronic playbackdevice 240. The media data captured by the video/audio recorder 220 mayalso be transmitted to the electronic playback device 240 as audio/videodata and may be encoded as discussed above prior to transmission anddecoded upon playback on the electronic playback device.

In the embodiment illustrated by FIG. 2A, the event being recorded bythe system is a hockey game and the subject of interest at the moment isa hockey puck HP being struck by a hockey stick HS. The non-contactmovement/vibration capture device 210 records the movement/vibration ofthe hockey puck HP as it is struck by the hockey stick HS. At the sametime, the contact sensor 216 in contact with the hockey puck HP measuresthe vibration generated by the striking of the hockey puck HP with thehockey stick HS. The data recorded by the movement/vibration capturedevice 210 and the contact sensor 216 are processed and transformed bythe transformer 230 into tactile effect data and transmitted to theelectronic playback device 240 alongside or interweaved with theaudio/video data. Upon receipt of the transmission, the electronicplayback device 240 may store the data in electronic memory for playbackat a later time, or may proceed to display the video images on thedisplay 244 and display a haptic effect HE corresponding to the sensedvibration with the haptic output device 242 so that the user holding theelectronic playback device 240 can feel the vibration associated withthe hockey stick HS hitting the hockey puck HP.

In an embodiment, the electronic playback device 240 may be theelectronic playback device 130 of FIG. 1, and may be in the form of amobile phone or tablet having the display 134, the speaker 136, and avibration device as the haptic output device 132 to provide the hapticeffect. In an embodiment, the haptic output device 242 may be part of anelectronic device that is constructed and arranged to be held by or beattached to the user of the device. For example, the haptic displaydevice 242 may be part of a mobile phone, such as a smartphone, atablet, a gamepad, a wearable device, such as a smartwatch, a wristband, an arm band, a head band, a head-mounted display, etc.

In order to maintain the egocentric video/audio recorder (i.e. camera104) perspective, the sensor 102 should track the camera 104point-of-view in a perceptually meaningful way. For example, if thecamera 104 is following a football during a football game, then thesensor 102 should be moved to maintain a synchronized perspective withthe camera 104. This synchronization between the camera 104 and thesensor 102 may be achieved in a variety of ways. In an embodiment,object tracking using markers or video based tracking techniques may beused. In an embodiment, the position and orientation of the camera 104and the sensor 102 may be manually controlled by the operator of thecamera 104 and sensor 102.

In an embodiment illustrated in FIG. 2B, servo matching of the positionand orientation of the video/audio recorder 220 (i.e. camera) to themovement/vibration capture device 210 (i.e. non-contact sensor) may beused. As illustrated, the system also includes a controller 250, a firstservomechanism (“servo”) 252 controlled by the controller 250 andconstructed and arranged to control the position and/or orientation ofthe contactless measurement device 212, a second servomechanism(“servo”) 254 controlled by the controller 250 and constructed andarranged to control the position and/or orientation of the trackingdevice 214, and a third servomechanism (“servo”) 256 controlled by thecontroller 250 and constructed and arranged to control the positionand/or orientation of the video/audio recorder 220. The controller 250may include a processor configured to execute three-dimensional imageanalysis software that provides the controller 250 with a subject markerthat can be translated into servo position commands for the servos 252,254, 256. In an embodiment, the three-dimensional image analysissoftware may comprise algorithms to determine the movement/vibrations ofthe subject and output a signal that may be transformed into the hapticeffect to be displayed to the user of the electronic playback device. Inan embodiment, the controller 250 may be commanded by a positiontracking system 260 that tracks the subject of interest inthree-dimensional space by using technology similar to existing beacon &triangulation systems. The position tracking system 260 provides asubject marker that may be translated into servo position commands forthe controller 250.

FIG. 3 illustrates an embodiment of data flow 300 from a non-contactsensor 310 and a video/audio recorder 320 to an electronic playbackdevice 330. As illustrated, data output by the sensor 310 undergoescalibration mapping at 312, is resampled at 314, and is encoded at 316using a format known to those skilled in the art, such as the formatdiscussed above, with an encoder. After the data is encoded, the encoderoutputs a signal 318 to be incorporated into media 340 that may bestreamed via the Internet, for example, for consumption as playbackmedia 350 on the electronic playback device 330. At the same time orsubstantially the same time, data from the video/audio recorder 320 isencoded at 322 with an encoder using a format known to those skilled inthe art, such as the format discussed above, to prepare an encodedsignal 324 to be incorporated into the media 320 that is streamed to theplayback device 330. The encoded signals 318, 324 may be synchronizedbefore being streamed (or otherwise delivered) to the playback device330, or the encoded signals 318, 324 may be synchronized locally on theplayback device 330 after being delivered to the playback device 330, asdiscussed above. The encoded signals 318, 324 may be streamed along-sideeach other or may be interleaved into a single signal.

FIG. 4 illustrates an embodiment of data flow 400 from more than onesensor, such as a non-contact sensor 410 measuring vibrations of asubject of interest and a contact sensor 420 that is in contact with thesubject of interest, and a video/audio recorder 430 recording thesubject of interest to an electronic playback device 440. In anembodiment, the sensor data may be processed to ensure calibration andsample rate consistency. Multiple sensor data channels may then becombined using various techniques, such as state estimation, mixing,averaging, etc. in order to generate a single haptic signal. In theembodiment illustrated in FIG. 4, the data output by the non-contactsensor 410 undergoes calibration mapping at 412 and is resampled at 414.The data output by the contact sensor 420 undergoes calibration mappingat 422 and is resampled at 424. The data after being resampled at 414and 424 is transformed at 416 for state estimation so that if the datain the two streams is at different frequencies, for example, the datasignals can be estimated and transformed into a single data signal forencoding at 418 by an encoder using a format known to those skilled inthe art, such as the format discussed above. In an embodiment, the datain the two streams may be transformed into a haptic output stream thatmay consist of one or more haptic data channels. The encoder outputs asignal 426 to be incorporated into media 436 that may be streamed viathe Internet, for example, for consumption as playback media 438 on theplayback device 440. At the same time or substantially the same time,data from the video/audio recorder 430 is encoded at 432 with an encoderusing a format known to those skilled in the art, such as the formatdiscussed above, to prepare an encoded signal 434 to be incorporatedinto the media 436 that is streamed to the playback device 440. Theencoded signals 426, 434 may be synchronized before being streamed orotherwise delivered to the playback device 440, and the encoded signals426, 434 may be synchronized locally on the playback device 440 afterbeing delivered to the playback device 440 but before being played backas the playback media 438, as discussed above. The encoded signals 426,434 may be streamed along-side each other or may be interleaved into asingle signal.

In both embodiments illustrated in FIGS. 3 and 4, the output of thefield based tactile sensing may include two distinct types of dataencoding, such as RAW master recording that includes the complete andun-mixed sensor data from the field sensor sources, and downmixrecording that includes the mixed and broadcast ready tactile track andis the final mixed product of all tactile sensors as broadcast. In someembodiments, the downmix recording may be available for each camera atthe event. Due to differences in sensor technologies, the RAW recordingsmay have a variety of sample rates, and the downmix may conform to astandard sample rate (e.g. 200 Hz) in order to facilitate downstreammixing and end-device rendering.

According to embodiments of the invention, the systems 100, 200described above may be media server driven. The media server may be acloud video streaming service such as, for example, NETFLIX®, abroadcast network-provided service such as, for example, NBC, a gamingconsole such as, for example Xbox by Microsoft, or any other type ofserver. The tactile effect data may be provided as a signal that iscommunicated, either wired or wirelessly, from the media server to theelectronic playback device 130, 240. The signal can be anything from adirect tactile effect stream or set of commands, to any abstractindication to the haptic output device 132, 242 that a number of actionsmust be performed, wherein one of the actions performed is the renderingof a haptic effect based on the tactile effect data.

According to embodiments of the invention, the systems 100, 200 may bePVD driven. The PVD may be configured to recognize the tactile effectdata signal and then send either the same tactile effect data signal, ora different signal transformed from the tactile effect data signal, thatultimately results in haptic effects being generated by the hapticoutput device 132, 242.

FIG. 5 illustrates a method 500 in accordance with an embodiment of theinvention. At 510, a video and/or audio is/are recorded by a recordingdevice, such as the video recorder 106 and/or audio recorder 108 or thevideo/audio recorder 220 described above. At 520, movement and/orvibrations are sensed by at least one non-contact sensor, such as themovement/vibration capture device 210 described above. In an embodiment,the movement/vibration data may be transformed into a tactile datastream. At 530, the video and the vibrations in the tactile data streamare synchronized using a processor, such as the processor 112 describedabove. At 540, the synchronized video and/or audio and vibrations areplayed back using an electronic playback device, such as the electronicplayback device 130 described above that includes the display 134, thespeaker 136 and the haptic output device 132, or the electronic playbackdevice 240 described above.

Embodiments of the invention described above capture movement and/orvibration of a subject of interest with the intent of using themovement/vibration capture to produce tactile effect data to be usedalongside video and/or audio data in a multi-media data stream. Themovement/vibration capture involves non-contact observation of thesubject of interest.

Embodiments of the invention described above provide the ability tocreate a live broadcast of an event that incorporates at least onenon-contact tactile sensor. Embodiments of the invention described abovemay be particularly useful for events that may have delicate or extremeenvironmental conditions not suitable for contact based tactile sensing.

Embodiments of the invention described above provide a camera-basedegocentric sound field that can be extended to enable both audio andtactile feedback to be captured and rendered as part of a sportsbroadcast or pre-production shoot. Embodiments of the inventiondescribed above provide a combination of on-subject and non-contacttactile sensing technologies that can be mixed to create the appropriateend-user experience for each camera's point of view.

The embodiments described herein represent a number of possibleimplementations and examples and are not intended to necessarily limitthe present disclosure to any specific embodiments. Instead, variousmodifications can be made to these embodiments as would be understood byone of ordinary skill in the art. Any such modifications are intended tobe included within the spirit and scope of the present disclosure andprotected by the following claims.

What is claimed is:
 1. A system comprising: a recorder configured torecord audio and/or video of a subject of interest and output arecording of the subject of interest; a non-contact sensor associatedwith the recorder, the non-contact sensor constructed and arranged tomeasure movement and/or vibration of the subject of interest fromsubstantially the same perspective and at the same time as the recorder;and a controller configured to transform the measured movement and/orvibration of the subject of interest measured by the non-contact sensorinto a tactile data stream for sending to a haptic display device forplayback with the recording of the subject of interest by the recorderand providing haptic effects corresponding to the measured movementand/or vibration to a user of the haptic display device insynchronization with the recording.
 2. The system according to claim 1,wherein the non-contact sensor comprises a laser vibrometer.
 3. Thesystem according to claim 1, wherein the non-contact sensor comprises atracking device.
 4. The system according to claim 1, wherein thenon-contact sensor comprises video stream analysis algorithms.
 5. Thesystem according to claim 1, wherein the recorder and the non-contactsensor are co-located.
 6. The system according to claim 1, furthercomprising a first servomechanism controlled by the controller andconstructed and arranged to move the non-contact sensor to a positionand/or orientation having substantially the same perspective as therecorder.
 7. The system according to claim 6, further comprising asecond servomechanism controlled by the controller and constructed andarranged to move the recorder, wherein the first servomechanism issynchronized with the second servomechanism so that movement of thenon-contact sensor is synchronized with movement of the recorder.
 8. Thesystem according to claim 1, further comprising a second sensorconstructed and arranged to be in contact with the subject beingrecorded by the recorder and to sense movement and/or vibration of thesubject, wherein the controller is further configured to combine datareceived from the non-contact sensor and the second sensor and transformthe combined data into the tactile data stream.
 9. The system accordingto claim 8, wherein the controller comprises a transformer constructedand arranged to mix sensors signals having different sampling ratesand/or sensitivity and output the transformed combined data.
 10. Amethod comprising: recording audio and/or video of a subject of interestwith a recorder; measuring movement and/or vibration of the subject ofinterest with a non-contact sensor associated with and havingsubstantially the same perspective as the recorder; and transforming,with a controller, the measured movement and/or vibration into a tactiledata stream for sending to a haptic display device for playback with therecording of the audio and/or video of the subject of interest andproviding haptic effects corresponding to the measured movement and/orvibration to a user of the haptic display device in synchronization withthe recording.
 11. The method according to claim 10, further comprisingmoving the non-contact sensor, with a first servomechanism controlled bythe controller, to a position having substantially the same perspectiveas the recorder as the recorder moves and changes perspective.
 12. Themethod according to claim 11, further comprising moving the recorder,with a second servomechanism controlled by the controller, wherein thefirst servomechanism is synchronized with the second servomechanism sothat movement of the non-contact sensor is synchronized with movement ofthe recorder.
 13. The method according to claim 10, further comprisingsensing movement and/or vibration of the subject of interest with asecond sensor in contact with the subject of interest; combining datareceived from the non-contact sensor and the second sensor; andtransforming the combined data into the tactile data stream.
 14. Themethod according to claim 13, further comprising transforming sensorsignals having different sampling rates and/or sensitivity andoutputting the transformed combined data.